1. Field of the Invention
The present invention relates to composite processing of an image using multi-viewpoint image data.
2. Description of the Related Art
Conventionally, when an image was captured by erroneous focus adjustment of a camera, it was necessary to recapture the image after performing focus adjustment again. Further, when it was desired to obtain focused images of a plurality of subjects the depths of which are different from each other, it was necessary to capture the image of each subject in focus a plurality of times.
In recent years, a technique called light field photography has been developed, that is capable of acquiring images from multiple viewpoints by adding a new optical element to the optical system and of adjusting the focus position by the later image processing (refocus).
By using this technique, there is an advantage that a failure in focus adjustment at the time of image capturing can be recovered by image processing because focus adjustment can be performed after image capturing. Further, there is also an advantage that it is possible to acquire a plurality of images focused on arbitrary subjects in an image from one captured image by changing the image processing method, and therefore, it is possible to reduce the number of times of image capturing.
In light field photography, the direction and intensity of a light beam that passes through each position (light field, hereinafter, referred to as “LF”) in a plurality of positions in the space are calculated from multi-viewpoint image data. Then, by using the information of the obtained LF, an image on the assumption that light passes through a virtual optical system and forms the image on a virtual sensor is calculated. By appropriately setting such a virtual optical system and a virtual sensor, refocus is also enabled. The mathematical nature and the mathematical foundation of LF have been discussed by R. NG etc. (R. NG, M. Levoy, M. Bredif, G. Duval, M. Horowitz, P. Hanrahan, “Light Field Photography with a Hand-held Plenoptic Camera” (Stanford Tech Report CTSR 2005-02, 2005). Hereinafter, processing to calculate image data obtained by a virtual sensor from multi-viewpoint image data is called composite processing.
As an image capturing technique for acquiring LF, a Plenoptic camera in which a microlens array is placed behind a main lens and a camera array in which compact cameras are arrayed are known. It is possible for both the techniques to combine image data on a virtual sensor after image capturing from the acquired LF. At this time, by changing the range of LF to be combined, it is made possible to adjust also the depth of field after image capturing.
As a method for combining images on a virtual sensor from LF, there is known a method in which acquired multi-viewpoint image data is subjected to projective transformation onto a virtual sensor, added and averaged (WO2008/050904).
However, the technique of WO2008/050904 has such a problem that an artifact occurs in a blur of a subject out of focus, and therefore, the blur is not smooth. As a method for solving the problem, there is known a method for reproducing a smooth blur by estimating the shape of a subject from multi-viewpoint image data and generating an image by performing tracing of a light beam having passed through a lens based on the information (Japanese Patent Laid-Open No. 2009-211254).
However, by the method of Japanese Patent Laid-Open No. 2009-211254, the light beam tracing is performed, and therefore, the burden of the processing is very heavy. Further, when the shape is estimated erroneously, the deterioration in image quality will be caused.